26 research outputs found
Anomalies in the peer-review system: A case study of the journal of High Energy Physics
Peer-review system has long been relied upon for bringing quality research to
the notice of the scientific community and also preventing flawed research from
entering into the literature. The need for the peer-review system has often
been debated as in numerous cases it has failed in its task and in most of
these cases editors and the reviewers were thought to be responsible for not
being able to correctly judge the quality of the work. This raises a question
"Can the peer-review system be improved?" Since editors and reviewers are the
most important pillars of a reviewing system, we in this work, attempt to
address a related question - given the editing/reviewing history of the editors
or re- viewers "can we identify the under-performing ones?", with citations
received by the edited/reviewed papers being used as proxy for quantifying
performance. We term such review- ers and editors as anomalous and we believe
identifying and removing them shall improve the performance of the peer- review
system. Using a massive dataset of Journal of High Energy Physics (JHEP)
consisting of 29k papers submitted between 1997 and 2015 with 95 editors and
4035 reviewers and their review history, we identify several factors which
point to anomalous behavior of referees and editors. In fact the anomalous
editors and reviewers account for 26.8% and 14.5% of the total editors and
reviewers respectively and for most of these anomalous reviewers the
performance degrades alarmingly over time.Comment: 25th ACM International Conference on Information and Knowledge
Management (CIKM 2016
The Effects of Gender Signals and Performance in Online Product Reviews
This work quantifies the effects of signaling gender through gender specific user names, on the success of reviews written on the popular amazon.com shopping platform. Highly rated reviews play an important role in e-commerce since they are prominently displayed next to products. Differences in reviews, perceived - consciously or unconsciously - with respect to gender signals, can lead to crucial biases in determining what content and perspectives are represented among top reviews. To investigate this, we extract signals of author gender from user names to select reviews where the author’s likely gender can be inferred. Using reviews authored by these gender-signaling authors, we train a deep learning classifier to quantify the gendered writing style (i.e., gendered performance) of reviews written by authors who do not send clear gender signals via their user name. We contrast the effects of gender signaling and performance on the review helpfulness ratings using matching experiments. This is aimed at understanding if an advantage is to be gained by (not) signaling one's gender when posting reviews. While we find no general trend that gendered signals or performances influence overall review success, we find strong context-specific effects. For example, reviews in product categories such as Electronics or Computers are perceived as less helpful when authors signal that they are likely woman, but are received as more helpful in categories such as Beauty or Clothing. In addition to these interesting findings, we believe this general chain of tools could be deployed across various social media platforms
A Review of the Role of Causality in Developing Trustworthy AI Systems
State-of-the-art AI models largely lack an understanding of the cause-effect
relationship that governs human understanding of the real world. Consequently,
these models do not generalize to unseen data, often produce unfair results,
and are difficult to interpret. This has led to efforts to improve the
trustworthiness aspects of AI models. Recently, causal modeling and inference
methods have emerged as powerful tools. This review aims to provide the reader
with an overview of causal methods that have been developed to improve the
trustworthiness of AI models. We hope that our contribution will motivate
future research on causality-based solutions for trustworthy AI.Comment: 55 pages, 8 figures. Under revie
Unsupervised ranking of clustering algorithms by INFOMAX.
Clustering and community detection provide a concise way of extracting meaningful information from large datasets. An ever growing plethora of data clustering and community detection algorithms have been proposed. In this paper, we address the question of ranking the performance of clustering algorithms for a given dataset. We show that, for hard clustering and community detection, Linsker's Infomax principle can be used to rank clustering algorithms. In brief, the algorithm that yields the highest value of the entropy of the partition, for a given number of clusters, is the best one. We show indeed, on a wide range of datasets of various sizes and topological structures, that the ranking provided by the entropy of the partition over a variety of partitioning algorithms is strongly correlated with the overlap with a ground truth partition The codes related to the project are available in https://github.com/Sandipan99/Ranking_cluster_algorithms